THEORETICAL ANALYSIS

A Combinatorial Approach to Neural Emergent Communication (2025.coling-main)

Challenge:	Recent advances in memory-efficient zeroth-order methods have limited their widespread adoption due to performance drops and a high risk of divergence.
Approach:	They propose a memory-efficient zeroth-order framework to improve performance and convergence of the MeZO methods by using only forward passes.
Outcome:	The proposed framework improves performance and convergence of the proposed methods on Roberta-Large and Llama-2-7B models.

Copied to clipboard

Zheyuan Zhang

Information Retrieval

Challenge:	Existing research on emergent communication uses the Lewis signaling game . however, the training data is limited and the messages are often ineffective .
Approach:	They propose a combinatorial algorithm to solve the symbolic complexity for classification, which is the minimum number of symbols in the message for successful communication.
Outcome:	The proposed algorithm increases the number of effective symbols in the emergent language.

RelWalk - A Latent Variable Model Approach to Knowledge Graph Embedding (2021.eacl-main)

Copied to clipboard

Danushka Bollegala, Huda Hakami, Yuichi Yoshida, Ken-ichi Kawarabayashi

Improving Gradient Trade-offs between Tasks in Multi-task Text Classification (2023.acl-long)

Challenge:	Existing methods for embedding entities and relations in knowledge graphs are heuristically motivated and theoretical understanding of such embeddables is underdeveloped.
Approach:	They extend the random walk model of word embeddings to Knowledge Graph Embeddings (KGEs) they propose a learning objective motivated by the theoretical analysis to learn KGEs from a given knowledge graph.
Outcome:	The proposed learning objective is motivated by the theoretical analysis to learn KGEs from a given knowledge graph.

Copied to clipboard

Heyan Chai, Jinhao Cui, Ye Wang, Min Zhang, Binxing Fang, Qing Liao

MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning (2024.acl-long)

Challenge:	Existing methods to mitigate task conflict problem are heuristics or gradient-based algorithms to achieve an arbitrary Pareto optimal trade-off among different tasks .
Approach:	They propose a gradient trade-off approach to mitigate the task conflict problem by using heuristics or gradient-based algorithms to achieve an arbitrary Pareto optimal trade- off among different tasks.
Outcome:	The proposed model can achieve an arbitrary Pareto optimal trade-off among different tasks near the main objective of multi-task text classification (MTC) it is found that training all tasks simultaneously yields degraded performance than learning them independently, leading to poor training.

Copied to clipboard

Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten Rijke, Zhumin Chen, Jiahuan Pei

UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities (2026.acl-long)

Challenge:	Large language models (LLMs) are the default paradigm for natural language processing (NLP) as the models’ scale and the diversity of tasks increase, fine-tuning becomes infeasible.
Approach:	They propose to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters and reduce their rank by 8 times .
Outcome:	The proposed model uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential.

Copied to clipboard

Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jinheon Baek, Sung Ju Hwang

Information Retrieval

Challenge:	Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries.
Approach:	They propose a framework to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities.
Outcome:	The proposed framework shows superiority over existing methods on 10 benchmarks of multiple modalities.

Few-Shot Novel Concept Learning for Semantic Parsing (2021.findings-emnlp)

Copied to clipboard

Soham Dan, Osbert Bastani, Dan Roth

Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation (2020.findings-emnlp)

Challenge:	Existing deep learning algorithms typically require thousands of examples to learn novel concepts.
Approach:	They propose an algorithm for learning novel concepts by representing them as programs over existing concepts.
Outcome:	The proposed approach outperforms end-to-end neural semantic parsers in a few-shot novel concept learning setting.

Copied to clipboard

Weijia Xu, Xing Niu, Marine Carpuat

Text Generation

Challenge:	Iterative Back-Translation and Dual Learning use different objectives and heuristic gradient approximation strategies, and have not been extensively compared.
Approach:	They propose a dual reconstruction objective that provides a unified view of Iterative Back-Translation and Dual Learning.
Outcome:	The proposed method is more effective than Dual Learning on German-English and Turkish-English tasks.

Two Examples are Better than One: Context Regularization for Gradient-based Prompt Tuning (2023.findings-acl)

Copied to clipboard

Hyeonmin Ha, Soyoung Jung, Jinsol Park, Minjoon Seo, Seung-won Hwang, Byung-Gon Chun

LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models (2024.naacl-long)

Challenge:	Prompting has gained tremendous attention as an efficient method for the adaptation of large-scale language models.
Approach:	They propose a regularization method that guides a prompt to produce a task context properly.
Outcome:	The proposed method improves prediction performance in a zero-shot in-context learning setting without demonstration examples for in-constitu learning.

Copied to clipboard

Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, Sinong Wang

Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models (2024.emnlp-main)

Challenge:	Currently, large language models (LLMs) train on short text segments due to the computational overhead quadratic in the input lengths of their Transformer architectures.
Approach:	They propose a method that allows LLMs pre-trained with 2K or 4K-long segments to generalize to up to 200M length inputs while retaining perplexity.
Outcome:	The proposed method achieves 2.7 decoding speed up and 7.5 memory saving over the original model.

Copied to clipboard

Siqi Wang, Zhengyu Chen, Bei Li, Keqing He, Min Zhang, Jingang Wang

Invisible Entropy: Towards Safe and Efficient Low-Entropy LLM Watermarking (2025.emnlp-main)

Challenge:	a study of large language models (LLMs) reveals the transferability and discrepancies of scaling laws between Dense and MoE models.
Approach:	They investigate the transferability and discrepancies of scaling laws between Dense Models and Mixture of Experts models.
Outcome:	The results show that the power-law scaling framework also applies to MoE Models .

Copied to clipboard

Tianle Gu, Zongqi Wang, Kexin Huang, Yuanqi Yao, Xiangliang Zhang, Yujiu Yang, Xiuying Chen

Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences (2022.naacl-main)

Challenge:	Existing methods to watermark low-entropy content are expensive and risky . IE reduces parameter size by 99% while achieving performance on par with state-of-the-art methods .
Approach:	They propose a logit-based watermarking paradigm that uses entropy-based features to predict whether the next token is high or low.
Outcome:	The proposed method reduces parameter size by 99% while achieving performance on par with state-of-the-art methods.

Copied to clipboard

Yifan Chen, Qi Zeng, Dilek Hakkani-Tur, Di Jin, Heng Ji, Yun Yang

Tchebycheff Procedure for Multi-task Text Classification (2020.acl-main)

Challenge:	Existing models for long sequences are not efficient due to the quadratic space and time complexity of the self-attention modules.
Approach:	They propose to reduce the quadratic complexity to linear (modulo logarithmic factors) by low-dimensional projection and row selection.
Outcome:	The proposed methods outperform transformer-based models with smaller time/space footprint on the Long Range Arena benchmark.

Copied to clipboard

Yuren Mao, Shuang Yun, Weiwei Liu, Bo Du

Information Extraction & Text Mining

Challenge:	Existing methods for text classification assume that multitask text classification problems are convex multiobjective optimization problems.
Approach:	They propose a Tchebycheff procedure to optimize multi-task classification problems without convex assumption.
Outcome:	The proposed method is able to find an arbitrary Pareto optimal solution in the PareTO set if the problem is convex, but excludes many Paret optimal solutions from its search scope.

Variance Sensitivity Induces Attention Entropy Collapse and Instability in Transformers (2025.emnlp-main)

Copied to clipboard

Jonghyun Hong, Sungyoon Lee

BanditMTL: Bandit-based Multi-task Learning for Text Classification (2021.acl-long)

Challenge:	Attention-based language models rely on the softmax function to convert attention logits into probability distributions, but this process can result in attention entropy collapse.
Approach:	They propose to use the softmax function to re-weight attention logits to create probability distributions, but this reweighting can lead to attention entropy collapse . they find that entropic-stable attention methods can prevent entrapment and enable more stable training by controlling or insensitive to variance of attention logit variance.
Outcome:	The proposed methods prevent attention entropy collapse and enable more stable training.

Copied to clipboard

Yuren Mao, Zekai Wang, Weiwei Liu, Xuemin Lin, Wenbin Hu

Rethinking DPO: The Role of Rejected Responses in Preference Misalignment (2025.findings-emnlp)

Challenge:	Existing methods to regularize task variance are unexplored in multi-task text classification.
Approach:	They propose a multi-task learning method based on adversarial multi-armed bandit to regularize the task variance by means of a mirror gradient ascent-descent algorithm.
Outcome:	The proposed method achieves state-of-the-art in multi-task text classification.

Copied to clipboard

Jae Hyeon Cho, JunHyeok Oh, Myunsoo Kim, Byung-Jun Lee

Challenge:	Existing methods that optimize for preferred responses struggle to meet primary objectives . rejected responses dominate the loss function, leading to suboptimal performance .
Approach:	They propose a method that bounds the influence of rejected responses while maintaining the original structure of DPO.
Outcome:	The proposed method outperforms existing algorithms in promoting preferred responses.

WatME: Towards Lossless Watermarking Through Lexical Redundancy (2024.acl-long)

Copied to clipboard

Liang Chen, Yatao Bian, Yang Deng, Deng Cai, Shuaiyi Li, Peilin Zhao, Kam-Fai Wong

Challenge:	Existing methods for text watermarking rely on arbitrary vocabulary partitioning during decoding, which compromises the availability of suitable tokens and significantly degrades the quality of responses.
Approach:	They propose a method that leverages linguistic prior knowledge of lexical redundancies in LLM vocabularies to seamlessly integrate watermarks.
Outcome:	The proposed approach preserves the expressive power of large language models while preserving watermark detectability.

Fuzzy Reasoning Chain (FRC): An Innovative Reasoning Framework from Fuzziness to Clarity (2025.findings-emnlp)

Copied to clipboard

Ping Chen, Xiang Liu, Zhaoxiang Liu, Zezhou Chen, Xingpeng Zhang, Huan Hu, Zipeng Wang, Kai Wang, Shuming Shi, Shiguo Lian

Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate (2025.naacl-long)

Challenge:	ambiguity, polysemy, or uncertainty remain significant challenges in natural language processing.
Approach:	They introduce a framework that integrates LLM semantic priors with continuous fuzzy membership degrees to create an explicit interaction between probability-based reasoning and fuzzy membership reasoning.
Outcome:	The proposed framework integrates semantic priors with continuous fuzzy membership degrees . it allows ambiguous inputs to be gradually transformed into clear and interpretable decisions .

Copied to clipboard

Xiaomeng Jin, Zhiqi Bu, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, Mingyi Hong

Attribution and Application of Multiple Neurons in Multimodal Large Language Models (2025.findings-emnlp)

Challenge:	Existing methods to remove unwanted knowledge from large language models are formulated as minimizing memorization through the loss of the model.
Approach:	They propose a normalized gradient difference algorithm that optimizes a forgetting objective and an automatic learning rate scheduler that allows for better control over the trade-off between the objectives.
Outcome:	The proposed method improves on TOFU and MUSE datasets while exhibiting stable training.

Copied to clipboard

Feiyu Wang, Ziran Zhao, Dong Yu, Pengyuan Liu

SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment (2025.findings-acl)

Challenge:	Existing methods to identify multimodal neurons in MLLMs are insufficiently understood . previous studies focused on identifying neurons corresponding to single-tokens .
Approach:	They propose a method to identify multimodal neurons in Transformer-based MLLMs . they introduce fuzzy set theory to model the complex relationship between neurons and semantic concepts .
Outcome:	The proposed method improves performance on the Visual Question Answering task.

Copied to clipboard

Wenqiao Zhu, Ji Liu, Lulu Wang, Jun Wu, Yulun Zhang

1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models (2025.findings-emnlp)

Challenge:	Existing methods for aligning Large Language Models with human values are limited and results of DPO are not resilient.
Approach:	They propose a self-guided direct preference optimization algorithm that incorporates a pilot term to steer the gradient flow during the optimization process.
Outcome:	The proposed method can generate human-preferred response up to 9.19% higher than previous methods.

Copied to clipboard

Zeliang Zong, Kai Zhang, Zheyang Li, Wenming Tan, Ye Ren, Yiyan Zhai, Jilin Hu

daDPO: Distribution-Aware DPO for Distilling Conversational Abilities (2025.findings-acl)

Challenge:	Low-rank approximation compresses the model by retaining its essential structure with minimal information loss.
Approach:	They propose a method that leverages the strengths of pruning and low-rank approximation for LLMs.
Outcome:	The proposed methods surpass the existing methods on LLaMA and Qwen2.5 models.

Copied to clipboard

Zhengze Zhang, Shiqi Wang, Yiqun Shen, Simin Guo, Dahua Lin, Xiaoliang Wang, Nguyen Cam-Tu, Fei Tan

Dialogue Systems & Conversational Agents

Challenge:	Knowledge distillation (KD) with Direct Preference Optimization (DPO) has emerged as a promising approach to enhance the conversational abilities of smaller models using a larger teacher model.
Approach:	They propose a framework that integrates the teacher's distributional information into DPO distillation while preserving theoretical guarantees.
Outcome:	The proposed framework outperforms existing methods in restoring performance for pruned models and enhancing smaller models within the same LLM family.

Defining a New NLP Playground (2023.findings-emnlp)

Copied to clipboard

Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi Fung, Charles Yu, Joel Tetreault, Eduard Hovy, Heng Ji

CED: Comparing Embedding Differences for Detecting Out-of-Distribution and Hallucinated Text (2024.findings-emnlp)

Challenge:	Recent explosion of performance of large language models (LLMs) has changed the field more abruptly and seismically than any other shift in the field’s 80 year history.
Approach:	They propose 20+ PhD-dissertation-worthy research directions to define a new NLP playground by combining theoretical analysis, new and challenging problems, learning paradigms and interdisciplinary applications.
Outcome:	The proposed research will cover theoretical analysis, new and challenging problems, learning paradigms and interdisciplinary applications.

Copied to clipboard

Hakyung Lee, Keon-Hee Park, Hoyoon Byun, Jeyoon Yeom, Jihee Kim, Gyeong-Moon Park, Kyungwoo Song

Two-Stage Fine-Tuning for Improved Bias and Variance for Large Pretrained Language Models (2023.acl-long)

Challenge:	Existing methods for detecting out-of-distribution (OOD) samples are limited due to their domain shift and computational limitations.
Approach:	They propose a training-free method to detect out-of-distribution (OOD) samples . they theoretically validate that specific auxiliary and oracle samples improve this distinction .
Outcome:	The proposed method improves the ability of pre-trained models to distinguish between ID and OOD samples in text classification and hallucination detection tasks.

Copied to clipboard

Lijing Wang, Yingya Li, Timothy Miller, Steven Bethard, Guergana Savova

Analyzing Transformers in Embedding Space (2023.acl-long)

Challenge:	Recent work challenges the bias-variance trade-off . large pretrained models can have large variance and overfit domain-specific data .
Approach:	They propose a bias-variance trade-off that implies learning methods need to balance complexity with data size to minimize under-fitting and over-fit.
Outcome:	The proposed method achieves strong results on SuperGLUE and clinical information extraction tasks.

Copied to clipboard

Guy Dar, Mor Geva, Ankit Gupta, Jonathan Berant

Hebbian-Guided Bi-Directional Rank Adaptation for Parameter-Efficient Fine-Tuning (2026.findings-acl)

Challenge:	Understanding Transformer-based models has attracted significant attention . a zero-pass approach is feasible for some parameters, and for two-layer attention networks .
Approach:	They propose a theoretical framework where parameters of a trained Transformer are interpreted by projecting them into the embedding space.
Outcome:	The proposed framework shows that pre-trained and fine-tuned models can be interpreted in embedding space.

Copied to clipboard

Xing Zhao, Liu Yang, Xi-Le Zhao, Yueqi Yang, Tianqi Jiang

Green & Sustainable NLP

Challenge:	Low-Rank Adaptation (LoRA) is a widely used method to fine-tune large language models . but its fixed-rank design cannot capture the varying importance across different layers .
Approach:	They propose a framework that bi-directionally reallocates low-rank capacity using Hebbian-inspired importance estimation.
Outcome:	Experiments show that HeBiRA improves performance over baselines.

FoE: Forest of Errors Makes the First Solution the Best in Large Reasoning Models (2026.acl-long)

Copied to clipboard

Kehan Jiang, Haonan Dong, Zhaolu Kang, Zhengzhou Zhu, Guojie Song

Reasoning

Challenge:	Recent Large Reasoning Models (LRMs) have demonstrated remarkable success in complex reasoning tasks.
Approach:	They propose a self-guided efficient reasoning framework that reduces FoE by pruning subs.
Outcome:	The proposed model outperforms eight competitive baselines while reducing token consumption by 37.7% 70.4%.

Data Pollination: An Emergent Ecological Process Driving AI Population Evolution (2026.acl-long)

Copied to clipboard

Shufang Xie, Qizhi Pei, Ang Lv, Jingyang Hu, Lijun Wu, Rui Yan

Revisiting Entropy in Reinforcement Learning for Large Reasoning Models (2026.findings-acl)

Challenge:	evidence from deployed systems suggests that language models interact through a shared data ecosystem.
Approach:	They propose to use data pollination to investigate stability dynamics under synthetic data training to investigate model collapse.
Outcome:	The proposed model can mitigate model collapse observed in recursive training, and improve performance across benchmarks.

Copied to clipboard

Renren Jin, Pengzhi Gao, Yuqi Ren, Zhuowen Han, Tongxuan Zhang, Wuwei Huang, Wei Liu, Jian Luan, Deyi Xiong

Reasoning

Challenge:	Reinforcement learning with verifiable rewards (RLVR) has emerged as a paradigm for enhancing the reasoning capabilities of large language models.
Approach:	They propose a positive-advantage reweighting approach that regulates model entropy by adjusting the loss weights assigned to tokens with positive advantages during RLVR training.
Outcome:	The proposed approach regulates model entropy by adjusting loss weights assigned to tokens with positive advantages during RLVR training while maintaining competitive performance.

SDD: Self-Degraded Defense against Malicious Fine-tuning (2025.acl-long)

Copied to clipboard

ZiXuan Chen, Weikai Lu, Xin Lin, Ziqian Zeng

Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling (2025.acl-long)

Challenge:	Open-source Large Language Models (LLMs) employ safety alignment methods to resist harmful instructions, but malicious fine-tuning can easily bypass these safeguards.
Approach:	They propose a framework to prevent malicious fine-tuning of large language models on harmful data by using alignment methods that encourage them to produce irrelevant responses to harmful prompts.
Outcome:	The proposed framework reduces the general capability of the LLM when malicious fine-tuning fails, rendering it incapable of following harmful instructions.

Copied to clipboard

Deng Qiyuan, Xuefeng Bai, Kehai Chen, Yaowei Wang, Liqiang Nie, Min Zhang